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Abstract: Some decision-making problems, i.e., multi-criteria decision analysis (MCDA) problems, 
require taking into account the attitudes of a large number of decision-makers and/or respondents. 
Therefore, an approach to the transformation of crisp ratings, collected from respondents, in grey 
interval numbers form based on the median of collected scores, i.e., ratings, is considered in this 
article. In this way, the simplicity of collecting respondents’ attitudes using crisp values, i.e., by 
applying some form of Likert scale, is combined with the advantages that can be achieved by using 
grey interval numbers. In this way, a grey extension of MCDA methods is obtained. The application 
of the proposed approach was considered in the example of evaluating the websites of tourism 
organizations by using several MCDA methods. Additionally, an analysis of the application of the 
proposed approach in the case of a large number of respondents, done in Python, is presented. The 
advantages of the proposed method, as well as its possible limitations, are summarized. 


Keywords: MCDA; grey interval numbers; group decision-making; Python 





1. Introduction 


Multi-criteria decision-making (MCDM), or multi-criteria decision analysis (MCDA), 
has so far been used for solving a large number of numerous different decision-making prob- 
lems [1-4]. Therefore, MCDA is dealing with solving complex real-world problems of the 
greatest interest to the organization that cannot be solved by conventional methods [5-8]. 
In due course of time, many multi-criteria analysis (MCA) methods were proposed, pri- 
marily due to the dynamic and rapid development of the field of operational research. The 
following can be mentioned as some of the most cited articles from this area: Hajkowicz 
and Collins [9], Hajkowicz and Higgins [10], Kaklauskas et al. [11], Kostreva et al. [12], and 
Belton and Vickers [13]. 

Besides this research, there are many studies in this area, such as: research and 
development project portfolio selection [14] (Mavrotas and Makryvelios, 2021), assessing 
national energy sustainability [15], energy consumption analysis of high-speed trains [16], 
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evaluation of transport emissions reduction policies [17], planning renewable energy use 
and carbon saving [18], and so forth. 

MCDA has also been used successfully for solving decision-making problems that 
are related to uncertainties or require a group decision-making approach for solving 
them [19-25]. As some examples of such approaches, the following can be mentioned: 
a grey absolute decision analysis [26], a multiple criteria decision analysis framework 
for the dispersed group [27], a fuzzy multi-criteria analysis [28-30], and collaborative 
decision-making in the multi-actor multi-criteria analysis [31]. 

From the aforestated, it is clear that some decision-making problems can be more 
adequately solved if a larger number of respondents take part in solving them. In such 
cases, the question that arises is how to transform the attitudes collected from respondents 
into group attitudes. 

The approach based on the use of a five-point Liker’s scale, or similar, can be men- 
tioned as one of the probably simplest approaches for collecting the respondents’ attitudes. 
So far, in numerous articles published in scientific journals, numerous approaches have 
been proposed for the transformation of individual attitudes acquired in this way into 
group attitudes. The results obtained, the advantages, as well as the weaknesses of these 
approaches, are also presented in these journals. 

In this article, an approach to the transformation of crisp ratings, collected from 
respondents, as grey interval numbers form based on the median of collected scores, i.e., 
ratings, is considered. Therefore, the article proposes the transformation of individual 
ratings collected from respondents into grey intervals with the aim of performing MCDA 
with minimal loss of information in relation to cases when crisp ratings are transformed 
into crisp group ratings. The application of the proposed approach was considered on 
the example of evaluating the websites of tourism organizations by using several MCDA 
methods, and also an analysis of the application of the proposed approach in the case of a 
large number of respondents was done in Python and described. Additionally, the main 
idea of the article was to propose a simple procedure for gathering respondent’s attitudes 
instead of a complex procedure that is sometimes difficult to understand by ordinary 
respondents /decision-makers who are not familiar with MCDM and fuzzy logic. 

Therefore, the rest of this article is organized as follows: In Section 2, some basic 
definitions about grey numbers are given, while a new approach is proposed in Section 3. In 
Section 4, a numerical illustration is presented in order to highlight the basic characteristics 
of the proposed approach, while in Section 5 an analysis of the obtained results is performed. 
Finally, conclusions are given at the end of the article. 


2. Preliminaries 


Definition 1. Grey number [32]. A grey number ®x is such a number whose exact value is 
unknown, but the range in which value can lie is known. 


Definition 2. Interval grey number [32]. An interval grey number is a grey number with a known 
lower bound x and upper bound x , but with the unknown value of x, and it is shown as follows: 


ex E [xx] = [x < x < x]. (1) 


Definition 3. The whitening function [33—35]. The whitening function transforms an interval 
grey number into a crisp number whose possible values lie between the bounds of the interval grey 
number. For the given interval grey number, the whitened value x,,) of interval grey number &x is 
defined as 

X(A) = (1 —A)x + Ax, (2) 


where € [0,1] denotes the whitening coefficient. 
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In the particular case A = 0.5, the whitened value becomes the mean of the interval 
grey number, as follows: 


xos) = 0.5(x +3). (3) 


3. The Newly Proposed Approach 
Suppose that the decision matrix is presented in the form 
D = [xi], (4) 
where: xi denotes the evaluation of alternative i to criterion j stated by the decision-maker 
k;i=1,...,m,and m denotes the number of alternatives; j = 1,... ,n, and n denotes the 
number of criteria; k = 1... K, and K denotes the number of decision-makers. 
Such a three-dimensional matrix can be transformed into a group two-dimensional 
matrix as follows: 
D = |x’; (5) 
with 


K 
xi = ( x =) /K. (6) 


Essentially, x';; denotes rating of alternative i to criterion j. Such defined x';; is actually 
the mean value of all assessments of the alternative i in relation to the criterion j. 

However, the matrix shown using Equation (4) can be also transformed into a grey 
group decision matrix, as follows: 


D = ([xjj,Xij]), (7) 
with 


Xij = ( L =) [n ; (8) 
k 


Ek- 


Xij = ( L =) /nt. (9) 
kekt 


In (8), k~ denotes the set of elements whose values are less than or equal to the median 
value of xi, and n~ denotes the number of elements in this set. Similarly, kt in (9) denotes 


a set of elements whose values are greater than or equal to the median value of xi and nt 
denotes the number of elements in this set. 


Example 


Let S be a sequence of 10 integers from interval [1, 5] and S = (1, 2, 3, 1,5, 3, 3, 1,4, 5). 

Then, the mean and median of S are as follows: mean = 2.80 and median = 3.00. The 
mean value of a number which is less than or equal to the median (1, 2, 3, 1, 3, 3, 1) is 
xı = 2.00 and the mean value of a number greater or equal to the median (3, 5, 3, 3, 4, 5) is 
Xy = 3.83. 

The mean value of such interval [2.00, 3.83], determined using Equation (3), is 2.915, 
and the distance between it and the mean is 2.915 — 2.80 = 0.115, that is in percent- 
ages 4.11%. 

The results obtained based on several sequences of randomly generated numbers from 
interval [1, 10] are shown in Table 1. The calculation was done in Python using the seed (1). 
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Table 1. Difference between the mean value of the sequence of numbers and the value obtained by 





the proposed approach. 

Sample Mean Median xI Xu i a. = ae 2 d (%) 

5 5.60 5.00 4.00 7.00 5.50 0.10 1.79 

10 7.10 7.50 5.40 8.80 7.10 0.00 0.00 

15 5.13 6.00 2.88 7.50 5.19 0.05 1.06 

20 6.50 7.00 5.08 8.00 6.54 0.04 0.64 

25 4.52 4.00 2.43 6.50 4.46 0.06 1.23 

50 5.20 5.00 3.07 7.14 5.11 0.09 1.81 

100 5.01 5.00 2.93 7.09 5.01 0.00 0.02 

150 5.16 5.00 3.05 6.91 4.98 0.18 3.45 


From Table 1, it can be seen that the difference between the mean value of the sequence 
of numbers and the value obtained by the proposed approach is not large. 


4. A Numerical Illustration 


In this section, the use of the proposed approach is presented in the case of evaluating 
websites of tourist organizations from Fastern Serbia. The evaluation was performed on 
the websites of 5 tourist organizations from the Timok frontier, or more precisely tourist 
organizations of the Municipalities of Boljevac, Bor, Majdanpek, Negotin and Kladovo 
(It is important to state that the order of municipalities does not correspond to the order of 
alternatives, because the aim of this article is not to favor any of the above-mentioned tourist 
organizations.). The evaluation is performed based on the following criteria: Visual design— 
Cı, Structure and navigability—C 2, Content—C3, Innovation—Cy4, Personalization—Cs. 

The evaluation was performed using ARAS [36], WASPAS [37], CoCoSo [38] and 
WISP [39] methods. In the first case, the evaluation was performed using ordinary MCDA 
methods and the mean value of the collected ratings, while in the first case, the evaluation 
was performed using the proposed approach. 

This illustration does not show all the possibilities that the proposed approach pro- 
vides in terms of analysis. The main goal was to compare the results obtained by applying 
the mean value of all assessments and the proposed approach, where the transformation of 
grey numbers was performed using Equation (3) and A = 0.5. 

The rating obtained from 10 respondents is shown in Tables 2-6. 


Table 2. Ratings of alternative A, in relation to the evaluation criteria obtained from 10 respondents. 





Ay I II Il IV V VI VII VIII IX x 
Cy 1 2 3 3 5 3 3 4 3 2 
C2 3 3 4 5 3 4 4 4 2 4 
C3 3 3 4 3 3 4 4 5 5 4 
C4 1 1 2 3 4 4 5 2 3 2 
C5 2 1 2 2 1 2 3 4 3 2 


Table 3. Ratings of alternative Az in relation to the evaluation criteria obtained from 10 respondents. 





Ap I I Ill IV v VI VI VI IX x 
Cy 3 5 4 4 4 4 5 3 2 2 
CG 5 5 4 5 4 5 5 4 3 4 
C3 5 4 4 4 3 4 5 3 3 3 
Cj 4 4 5 4 2 2 5 3 5 3 
C; 4 4 5 3 4 3 4 3 2 4 
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Table 4. Ratings of alternative A3 in relation to the evaluation criteria obtained from 10 respondents. 





A3 I II Il IV vV VI VII VIII IX x 
Cy 1 2 2 2 2 2 2 3 2 1 
C2 3 5 5 4 2 2 3 4 4 4 
C3 1 4 4 2 2 2 4 3 2 2 
C4 1 1 3 3 1 1 4 2 1 1 
C5 3 5 5 3 4 4 3 4 4 4 





Ag I II Il IV vV VI VII VIII IX X 
Cı 4 4 4 5 4 4 4 4 5 4 
Co 5 5 4 5 3 3 4 5 5 3 
C3 5 5 4 4 3 3 4 5 5 5 
C4 4 4 5 5 5 3 5 5 4 3 
C5 4 4 5 3 4 4 3 5 4 4 


Table 6. Ratings of alternative As in relation to the evaluation criteria obtained from 10 respondents. 





Ag I II Ill IV V VI VII VIII IX x 
Cy 4 3 5 5 5 3 5 3 4 3 
C2 4 4 4 5 4 3 5 4 4 4 
C3 5 4 4 4 3 3 4 4 3 5 
C4 4 4 5 3 2 4 3 5 3 3 
C5 3 4 4 4 4 3 4 5 4 4 


The group decision matrix, formed on the basis of the responses of all respondents, 
is shown in Table 7. The elements of this matrix represent the mean value of the ratings 


obtained from the respondents. 


Table 7. Group decision-making matrix. 





Criteria Alternatives Cy C2 C3 C4 C5 
Ay 2.90 3.60 3.80 2.70 2.20 
Ad 3.60 4.40 3.50 3.70 3.60 
A3 1.90 3.60 2.60 1.80 3.90 
Ag 4.20 4.20 4.30 4.30 4.00 
As 4.00 4.10 3.90 3.60 3.90 


A similar decision matrix is shown in Table 8, where the elements of that matrix 


represent the median of ratings obtained from the respondents. 


Table 8. The median of ratings obtained from the respondents. 





Criteria Alternatives Cy C2 C3 C4 C5 
Ay 2.96 3.81 3.92 2.70 2.11 
Ao 3.79 4.40 3.50 3.82 3.81 
A3 1.95 3.79 2.31 1.40 3.96 
A4 4.10 4.20 4.30 4.30 4.00 
A5 4.00 4.05 3.96 3.60 3.95 


The results of the evaluation performed using ordinary ARAS, WASPAS, CoCoSo and 
WISP methods, weighting vector w; = (0.25, 0.24, 0.22, 0.20, 0.10), and the data from Table 7, 
are shown in Table 9. 
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Table 9. Ranking of alternatives using ordinary ARAS, WASPAS, CoCoSo and WISP methods. 





ARAS WASPAS CoCoSo WISP 
Alternatives Si Rank Si Rank Si Rank Si Rank 
Ay 0.73 4 0.73 4 1.80 4 0.87 4 
A2 0.88 3 0.88 3 2.17 3 0.95 3 
A3 0.61 5 0.60 5 1.49 5 0.81 5 
A4 0.99 1 0.99 1 2.43 1 1.00 1 
A5 0.92 2 0.92 2 2.25 2 0.96 2 


In the second case, based on data from Table 8 as well as ratings from Tables 2—6, a 
grey decision matrix was formed as shown in Table 10. 


Table 10. Grey group decision-making matrix. 





Criteria Alternatives Cy C2 C3 C4 C5 
Ay [2.50, 3.43] [3.44, 4.17] [3.50, 4.33] —«*[1.60, 3.80] [1.71, 2.50] 
Ag [3.25, 4.33] [3.80, 5.00] [2.80, 4.20] [3.14, 4.50] [3.44, 4.17] 
A3 [1.78, 2.13]  [3.25,4.33] [1.83,2.78] [1.00,1.80] [3.62, 4.29] 
A4 [4.00, 4.20] [3.40, 5.00] [3.60, 5.00] [3.60, 5.00] [3.75, 4.25] 
As [3.33, 4.67] [3.88, 4.22] [3.63, 4.29] [2.80, 4.40] [3.78, 4.13] 


The evaluation results generated using the grey ARAS, WASPAS, CoCoSo and WISP 
methods, and the data from Table 10, are shown in Table 11. It should be noted again that 
the grey numbers from Table 10 were transformed into crisp values, using Equation (3) and 
A = 0.5, before the evaluation. 


Table 11. Ranking of alternatives using grey WS, WP, WASPAS and CoCoSo methods. 





ARAS WASPAS CoCoSo WISP 
Alternatives Si Rank Si Rank Si Rank Si Rank 
Ay 0.76 4 0.46 4 1.90 4 0.74 4 
A? 0.91 3 0.55 3 2.29 3 0.89 3 
A3 0.59 5 0.36 5 1.47 5 0.62 5 
A4 0.99 1 0.60 1 2.48 1 0.99 1 
A5 0.92 2 0.56 2 2.32 2 0.92 2 


From Tables 9 and 11, it can be seen that differences in ranking orders of alternatives 
achieved on the basis of the mean value of all assessments and the proposed approach 
were not observed. Of course, it should be reiterated here that the proposed approach 
provides significantly greater opportunities in terms of analyzing various scenarios, such 
as pessimistic or optimistic. 


5. Analysis and Discussion 


In order to verify the proposed approach, this section presents the results of the 
evaluation based on the assessments of a number of virtual respondents. For easier 
evaluation, the scores were generated as random numbers from the interval [1, 10], using 
a program written in the Python programming language, in which all calculations were 
also performed. In this analysis, random numbers are generated with the seed (1). The 
results obtained on the basis of series of 10, 50, 100 and 150 virtual respondents are shown 
in Tables 12-20. The weighting vector w; = (0.2, 0.2, 0.2, 0.2, 0.2) is used in this evaluation. 

The calculation details obtained on the basis of 10 virtual respondents are shown in 
Tables 12 and 13. As can be seen from Tables 12 and 13, in this case, the same ranking 
orders are obtained by applying all methods and approaches. 
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Table 12. Ranking of alternatives on the basis of 10 virtual respondents and crisp approach. 
WS WP WASPAS CoCoSo WISP 
Alternatives Si Rank P; Rank Qi Rank Ki Rank Si Rank 
Ay 0.83 5 0.75 5 0.75 5 1.77 5 0.59 5 
Az 1.00 1 0.91 1 0.91 1 2.16 1 1.00 1 
A3 0.98 2 0.87 2 0.88 2 2.09 2 0.88 2 
A4 0.84 4 0.75 4 0.76 4 1.79 4 0.61 4 
A5 0.89 3 0.80 3 0.81 3 1.91 3 0.71 3 
Table 13. Ranking of alternatives on the basis of 10 virtual respondents and the proposed grey approach. 
WS WP WASPAS CoCoSo WISP 
Alternatives Si Rank P; Rank Qi Rank Ki Rank Si Rank 
Ay 0.82 5 0.74 5 0.75 5 1.78 5 0.58 5 
Ay 1.00 1 0.92 1 0.92 1 2.18 1 1.00 1 
A3 0.95 2 0.84 2 0.86 2 2.04 2 0.80 2 
Ag 0.83 4 0.75 4 0.75 4 1.79 4 0.59 4 
As 0.88 3 0.80 3 0.81 3 1.91 3 0.69 3 
Table 14. Ranking of alternatives on the basis of 50 virtual respondents and crisp approach. 
WS WP WASPAS CoCoSo WISP 
Alternatives Si Rank P; Rank Si Si Ki Rank Si Rank 
Ay 0.97 3 0.94 3 0.94 3 1.90 3 0.91 3 
Ag 0.92 4 0.89 4 0.89 4 1.80 4 0.78 4 
A3 1.00 1 0.97 1 0.97 1 1.97 1 1.00 1 
Ag 0.91 5 0.88 5 0.89 5 1.79 5 0.77 5 
As 0.97 2 0.94 2 0.95 2 1.91 2 0.91 2 
Table 15. Ranking of alternatives on the basis of 50 virtual respondents and the proposed grey approach. 
WS WP WASPAS CoCoSo WISP 
Alternatives Si Rank P; Rank Qi Rank Ki Rank S; Rank 
Ay 0.98 2 0.94 2 0.94 2 1.90 2 0.94 2 
Ag 0.92 5 0.89 5 0.89 5 1.79 5 0.80 5 
A3 1.00 1 0.96 1 0.96 1 1.94 1 1.00 1 
Ag 0.93 4 0.89 4 0.89 4 1.81 4 0.82 4 
As 0.97 3 0.93 3 0.93 3 1.87 3 0.90 3 
Table 16. Ranking orders of alternatives obtained on the basis of 50 virtual respondents. 
Crisp Grey Approach 
Alternatives WS WP WASPAS CoCoSo WISP WS WP WASPAS CoCoSo WISP 
Ay 3 3 3 3 3 2 2 2 2 2 
Ag 4 4 4 4 4 5 5 5 5 5 
A3 1 1 1 1 1 1 1 1 1 1 
A4 5 5 5 5 5 4 4 4 4 4 
As 2 2 2 2 2 3 3 3 3 3 
Table 17. Ranking orders of alternatives obtained on the basis of 100 virtual respondents. 
Crisp Grey Approach 
Alternatives WS WP WASPAS CoCoSo WISP WS WP WASPAS CoCoSo WISP 
Ay 4 4 4 4 4 4 4 4 4 4 
Ag 3 3 3 3 3 2 2 2 2 2 
A3 2 2 2 2 2 3 3 3 3 3 
A4 1 1 1 Í 1 1 1 1 1 1 
A5 5 5 5 5 5 5 5 5 5 5 
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Table 18. Ranking of alternatives on the basis of 150 virtual respondents and crisp methods. 
WS WP WASPAS CoCoSo WISP 
Alternatives Si Rank P; Rank Si Si Ki Rank Si Rank 
Ay 0.993 2 0.960 2 0.961 2 1.840 2 0.977 2 
Ad 0.975 3 0.944 3 0.944 3 1.808 3 0.927 3 
A3 0.971 5 0.938 5 0.939 5 1.799 5 0.914 5 
A4 1.000 1 0.968 1 0.968 1 1.854 1 1.000 1 
As 0.973 4 0.942 4 0.942 4 1.805 4 0.923 4 
Table 19. Ranking of alternatives on the basis of 150 virtual respondents and grey methods. 
WS WP WASPAS CoCoSo WISP 
Alternatives Si Rank P; Rank Qi Rank Ki Rank Si Rank 
Aj 0.988 2 0.954 2 0.955 2 1.806 2 0.965 2 
Ao 0.988 3 0.953 3 0.954 3 1.805 3 0.963 3 
A3 0.987 4 0.952 4 0.953 4 1.804 4 0.960 4 
A4 1.000 1 0.966 1 0.966 1 1.828 1 1.000 1 
As 0.985 5 0.952 5 0.952 5 1.801 5 0.958 5 
Table 20. Ranking orders of alternatives obtained on the basis of 150 virtual respondents. 
Crisp Grey Approach 
Alternatives WS WP WASPAS  CoCoSo WISP WS WP WASPAS CoCoSo WISP 
Aj 2 2 2 2 2 2 2 2 2 2 
A? 3 3 3 3 3 3 3 3 3 3 
A3 5 5 5 5 5 4 4 4 4 4 
A4 1 1 1 1 1 1 1 1 1 1 
As 4 4 4 4 4 5 5 5 5 5 
The calculation details obtained on the basis of 50 virtual respondents are shown in 
Tables 14-16. In this case, there were some discrepancies in the order of the second and 
third-placed alternatives, which can be clearly seen in Figure 1. 
6 
= 
4 
T 
2 
1 
j WS WP WASPAS CoCoS WISP 
Ws WP WASPAS. CoCoSo WISP : : dal i 
Grey Grey - Grey Grey Grey 
— A1 3 3 3 3 3 2 2 2 2 2 
om A2 4 4 4 4 4 5 5 5 5 5 
A3 1 1 al T 1 1 ul 1 1 1 
oo A4 5 5 5 5 5 4 4 4 4 4 
— A5 2 2 2 2 2 3 3 3 3 3 


Figure 1. Ranking orders of alternatives obtained on the basis of 50 virtual respondents. 


—— A] ~ A? 


A3 qo A4 ~ A5 
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It can be seen from Tables 14 and 15 that differences between second-placed and third- 
placed alternatives are not high, which is why it can be expected that the same ranking 
order of alternatives could be obtained by using another weight vector. 

Ranking orders of alternatives, obtained on the basis of 100 virtual respondents, are 
shown in Tabe 17, and presented in Figure 2. This case is similar to the previous one. 











O e NU FUN DD 


Ws WP WASPAS. CoCoSo WISP si a ee wee : pan 
=—Ai| 4 4 4 4 4 4 4 4 4 4 
=| 3 3 3 3 3 2 2 2 2 2 
——A3| 2 2 2 2 2 3 3 3 3 3 
—A4] 1 1 1 1 1 1 1 1 1 1 
=^; 5 5 5 5 5 5 5 5 5 5 


ee AD = AD ~ A3 ~AG eee AD 


Figure 2. Ranking orders of alternatives obtained on the basis of 100 virtual respondents. 


From Table 17 and Figure 2 it can be observed that in this case, the differences occur 
only in the case of the second and third-placed alternatives. 

Ranking orders of alternatives that arise from 150 virtual respondents are arranged in 
Tables 18-20 and presented in Figure 3. 











6 
5 
CO 
3 
2 
1 
0 
ws WP WASPAS. CoCoSo WISP ae mi a ree ped 
—si| 2 2 2 2 2 2 2 2 2 2 
—A2| 3 3 3 3 3 3 3 3 3 3 
am A3| 5 5 5 5 5 4 4 4 4 4 
—A4] 1 1 1 1 1 1 1 1 1 1 
=A; 4 4 4 4 4 5 5 5 5 5 


ee A A? ~ A3 ce Ad eee AD 


Figure 3. Ranking orders of alternatives obtained on the basis of 150 virtual respondents. 


Table 20 and Figure 3 clearly show that the alternative A4 is best ranked according to 
all methods, with all crisp methods gave the same order of ranking Aq A1, A2, A5, A3, while 
the proposed grey approach gave the following rankings order A4, A1, A2, A3, A5. However, 
from Table 20 it is observable that there are very small differences in overall performance 
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between the second-placed, third-placed and fourth-placed alternatives, which is why it 
can be expected that different ranking orders of alternatives could be obtained by using 
another weighting vector. 


6. Conclusions 


The advantages of using grey instead of crisp numbers in multi-criteria decision anal- 
ysis have been considered and proven in a number of previously published studies. One 
of the advantages which should be emphasized using grey numbers is the possibility of 
considering various scenarios, such as: pessimistic, realistic, and optimistic. The proposed 
approach allows the transformation of crisp ratings, collected by employing surveys based 
on the use of the Likert scale, into grey numbers and thus considering different scenarios. 
The proposed approach may be suitable when it is necessary to collect and analyze the 
realistic attitudes of a larger number of respondents. Moreover, the proposed transforma- 
tion enables greater robustness and further possibility of analysis and consideration of 
different scenarios. 

The results of the website evaluation based on the mean value of the ratings obtained 
from all respondents and the proposed approach did not indicate a difference in the 
ranking orders of alternatives. However, the results of the conducted analysis indicate 
that differences may arise between the two approaches, especially in the case of the lower- 
ranked alternatives. 

Some differences in the results are expected because the proposed approach is not a 
substitute for applying the mean value of the scores obtained from all respondents, but an 
approach that further allows the possibility of analysis. Certain differences in the ranking 
results using the newly proposed approach and applying the mean of the scores obtained 
in all respondents can be cited as a weakness of this approach. 

Finally, consideration of the transformation of a larger number of crisp ratings into 
corresponding triangular fuzzy numbers or interval-valued triangular fuzzy numbers 
can be mentioned as one of the possible directions for the further development of the 
proposed approach. 


Author Contributions: Conceptualization, P.S.S., D.K., V.N.K. and G.P.; methodology, D.K., D.S. and 
FS.; validation, A.U.; investigation, A.U.; data curation, G.P.; writing—original draft preparation, 
D.S., V.N.K. and M.S.; writing—review and editing, P.S.S. and M.S.; supervision, D.K.; funding 
acquisition, F.S. All authors have read and agreed to the published version of the manuscript. 


Funding: The research presented in this article was done with the financial support of the Ministry 
of Education, Science and Technological Development of the Republic of Serbia, within the funding 
of the scientific research work at the University of Belgrade, Technical Faculty in Bor, according to 
the contract with registration number 451-03-9/2021-14/200131. 


Conflicts of Interest: The authors declare no conflict of interest. 


References 

1. Ulutaş, A.; Stanujkic, D.; Karabasevic, D.; Popovic, G.; Zavadskas, E.K.; Smarandache, F.; Brauers, W.K. Developing of a Novel 
Integrated MCDM MULTIMOOSRAL Approach for Supplier Selection. Informatica 2021, 32, 145-161. [CrossRef] 

2.  Karabasevic, D.; Radanov, P.; Stanujkic, D.; Popovic, G.; Predic, B. Going green: Strategic evaluation of green ICT adoption in the 
textile industry by using bipolar fuzzy Multimoora method. Ind. Text. 2021, 72, 3-10. [CrossRef] 

3. Jaukovic Jocic, K.; Jocic, G.; Karabasevic, D.; Popovic, G.; Stanujkic, D.; Zavadskas, E.K.; Thanh Nguyen, P. A novel integrated 
piprecia—interval-valued triangular fuzzy aras model: E-learning course selection. Symmetry 2020, 12, 928. [CrossRef] 

4. Karabagevi¢, D.; Stanujki¢, D.; Zavadskas, E.K.; Stanimirović, P.; Popovic, G.; Predić, B.; Ulutaş, A. A Novel Extension of the 
TOPSIS Method Adapted for the Use of Single-Valued Neutrosophic Sets and Hamming Distance for E-Commerce Development 
Strategies Selection. Symmetry 2020, 12, 1263. [CrossRef] 

5. Amiri, M.; Hashemi-Tabatabaei, M.; Ghahremanloo, M.; Keshavarz-Ghorabaee, M.; Zavadskas, E.K.; Antucheviciene, J. A novel 
model for multi-criteria assessment based on BWM and possibilistic chance-constrained programming. Comput. Ind. Eng. 2021, 
156, 107287. [CrossRef] 

6. Lu, K,; Liao, H.; Zavadskas, E.K. An overview of fuzzy techniques in supply chain management: Bibliometrics, methodologies, 


applications and future directions. Technol. Econ. Dev. Econ. 2021, 27, 402-458. [CrossRef] 


Axioms 2021, 10, 124 11 of 12 





28. 


29. 


30. 


31. 


32. 


33. 


34. 


Stanujkic, D.; Zavadskas, E.K.; Liu, S.; Karabasevic, D.; Popovic, G. Improved OCRA method based on the use of interval grey 
numbers. J. Grey Syst. 2017, 29, 49—60. 

Popovic, G.; Stanujkic, D.; Brzakovic, M.; Karabasevic, D. A multiple-criteria decision-making model for the selection of a hotel 
location. Land Use Policy 2019, 84, 49-58. [CrossRef] 

Hajkowicz, S.; Collins, K. A review of multiple criteria analysis for water resource planning and management. Water Resour. 
Manag. 2007, 21, 1553-1566. [CrossRef] 

Hajkowicz, S.; Higgins, A. A comparison of multiple criteria analysis techniques for water resource management. Eur. J. Oper. 
Res. 2008, 184, 255-265. [CrossRef] 

Kaklauskas, A.; Zavadskas, E.K.; Raslanas, S. Multivariant design and multiple criteria analysis of building refurbishments. 
Energy Build. 2005, 37, 361-372. [CrossRef] 

Kostreva, M.M.; Ogryczak, W.; Wierzbicki, A. Equitable aggregations and multiple criteria analysis. Eur. J. Oper. Res. 2004, 158, 
362-377. [CrossRef] 

Belton, V.; Vickers, S.P. Demystifying DEA—A visual interactive approach based on multiple criteria analysis. J. Oper. Res. Soc. 
1993, 44, 883-896. 

Mavrotas, G.; Makryvelios, E. Combining multiple criteria analysis, mathematical programming and Monte Carlo simulation to 
tackle uncertainty in Research and Development project portfolio selection: A case study from Greece. Eur. J. Oper. Res. 2021, 291, 
794-806. [CrossRef] 

Phillis, A.; Grigoroudis, E.; Kouikoglou, V.S. Assessing national energy sustainability using multiple criteria decision analysis. 
Int. J. Sustain. Dev. World Ecol. 2021, 28, 18-35. [CrossRef] 

Peng, Y.; Zhang, H.; Wang, T.; Yang, M.; Wang, K.; Meng, W.; Wang, D. Energy consumption analysis and multiple-criteria 
evaluation of high-speed trains with different marshaled forms in China. Sci. Total Environ. 2021, 759, 143678. [CrossRef] 
Hasan, M.A.; Chapman, R.; Frame, D.J. Acceptability of transport emissions reduction policies: A multi-criteria analysis. Renew. 
Sustain. Energy Rev. 2020, 133, 110298. [CrossRef] 

Ahmed, A.; Sutrisno, S.W.; You, S. A two-stage multi-criteria analysis method for planning renewable energy use and carbon 
saving. Energy 2020, 199, 117475. [CrossRef] 

Huang, H.; Mommens, K.; Lebeau, P.; Macharis, C. The Multi-Actor Multi-Criteria Analysis (MAMCA) for Mass-Participation 
Decision Making. In International Conference on Decision Support System Technology; Springer International Publishing: Cham, 
Switzerland, 2021; pp. 3-17. 

Salo, A.; Hämäläinen, R.; Lahtinen, T. Multicriteria Methods for Group Decision Processes: An Overview. In Handbook of Group 
Decision and Negotiation; Kilgour, D.M., Eden, C., Eds.; Springer Science & Business Media: Berlin/ Heidelberg, Germany, 2021. 
Tao, R.; Liu, Z.; Cai, R.; Cheong, K.H. A dynamic group MCDM model with intuitionistic fuzzy set: Perspective of alternative 
queuing method. Inf. Sci. 2021, 555, 85-103. [CrossRef] 

Hsu, W.C,J.; Liou, J.J.; Lo, H.W. A group decision-making approach for exploring trends in the development of the healthcare 
industry in Taiwan. Decis. Support Syst. 2021, 141, 113447. [CrossRef] 

Zhan, J.; Zhang, K.; Wu, W.Z. An investigation on Wu-Leung multi-scale information systems and multi-expert group decision- 
making. Expert Syst. Appl. 2021, 170, 114542. [CrossRef] 

Liu, S.; Yu, W.; Chan, F.T.; Niu, B. A variable weight-based hybrid approach for multi-attribute group decision making under 
interval-valued intuitionistic fuzzy sets. Int. J. Intell. Syst. 2021, 36, 1015-1052. [CrossRef] 

Biswas, S.; Pamucar, D. Facility location selection for b-schools in indian context: A multi-criteria group decision based analysis. 
Axioms 2020, 9, 77. [CrossRef] 

Javed, S.A.; Mahmoudi, A.; Liu, S. Grey Absolute Decision Analysis (GADA) Method for multiple criteria group decision-making 
under uncertainty. Int. J. Fuzzy Syst. 2020, 22, 1073-1090. [CrossRef] 

Carneiro, J.; Martinho, D.; Alves, P.; Conceição, L.; Marreiros, G.; Novais, P. A multiple criteria decision analysis framework for 
dispersed group decision-making contexts. Appl. Sci. 2020, 10, 4614. [CrossRef] 

Al Asbahi, A.A.M.H.; Fang, Z.; Chandio, Z.A.; Tunio, M.K.; Ahmed, J.; Abbas, M. Assessing barriers and solutions for Yemen 
energy crisis to adopt green and sustainable practices: A fuzzy multi-criteria analysis. Environ. Sci. Pollut. Res. 2020, 27, 
36765-36781. [CrossRef] 

Musaad, O.A.S.; Zhuo, Z.; Musaad, O.A.O.; Ali Siyal, Z.; Hashmi, H.; Shah, S.A.A. A fuzzy multi-criteria analysis of barriers and 
policy strategies for small and medium enterprises to adopt green innovation. Symmetry 2020, 12, 116. [CrossRef] 

Durmi¢, E.; Stević, Z.; Chatterjee, P.; Vasiljević, M.; Tomašević, M. Sustainable supplier selection using combined FUCOM—Rough 
SAW model. Rep. Mech. Eng. 2020, 1, 34-43. [CrossRef] 

Huang, H.; De Smet, Y.; Macharis, C.; Doan, N.A.V. Collaborative decision-making in sustainable mobility: Identifying possible 
consensuses in the multi-actor multi-criteria analysis based on inverse mixed-integer linear optimization. Int. J. Sustain. Dev. 
World Ecol. 2021, 28, 64-74. [CrossRef] 

Deng, J.L. Introduction to Grey System Theory. J. Grey Syst. 1989, 1, 1-24. 

Liu, S.F.; Yang, YJ.; Forrest, J.Y.L. Grey Data Analysis: Methods, Models and Applications; Springer: Berlin/ Heidelberg, Germany, 
2016. 

Liu, S.F.; Forrest, J.Y.L. Grey Systems: Theory and Applications; Springer: Berlin/ Heidelberg, Germany, 2010. 


Axioms 2021, 10, 124 12 of 12 





35. 


36. 


37. 


38. 


39. 


Liu, S.; Rui, H.; Fang, Z.; Yang, Y.; Forrest, J. Explanation of terms of grey numbers and its operations. Grey Syst. Theory Appl. 
2016, 6, 436-441. [CrossRef] 

Turskis, Z.; Zavadskas, E.K. A novel method for multiple criteria analysis: Grey additive ratio assessment (ARAS-G) method. 
Informatica 2010, 21, 597-610. [CrossRef] 

Zavadskas, E.K.; Turskis, Z.; Antucheviciene, J. Selecting a Contractor by Using a Novel Method forMultiple Attribute Analysis: 
Weighted Aggregated SumProduct Assessment with Grey Values (WASPAS-G). Stud. Inform. Control. 2015, 24, 141-150. [CrossRef] 
Yazdani, M.; Wen, Z.; Liao, H.; Banaitis, A.; Turskis, Z. A grey combined compromise solution (COCOSO-G) method for supplier 
selection in construction management. J. Civ. Eng. Manag. 2019, 25, 858-874. [CrossRef] 

Stanujkic, D.; Popovic, G.; Karabasevic, D.; Meidute-Kavaliauskiene, I.; Ulutas, A. An Integrated Simple Weighted Sum Product 
Method—WISP. IEEE Trans. Eng. Manag. 2021, 1-12. [CrossRef] 


